Skip to content

ci(security): audit the latest release line's pinned runtime on the daily cron (BACKLOG #1081) - #267

Merged
wshallwshall merged 6 commits into
mainfrom
claude/released-line-audit
Aug 7, 2026
Merged

ci(security): audit the latest release line's pinned runtime on the daily cron (BACKLOG #1081)#267
wshallwshall merged 6 commits into
mainfrom
claude/released-line-audit

Conversation

@wshallwshall

Copy link
Copy Markdown
Collaborator

Adds a released-line-audit job to security.yml's existing daily cron. It audits the LATEST RELEASE
TAG's docker/locks/requirements-core.lock.

The required pip-audit job audits the checked-out tree, so on a schedule it audits main. This
covers the different window: the release lag between a fix landing on main and a release carrying it.

Retraction carried in the commit and in #1081

An earlier framing of mine -- "advisories after a tag are never re-evaluated" -- was WRONG.
CVE-2026-69247 was caught same-day by the required pip-audit gate (ac87246f). This job is not a
second detector for that case.
It covers the release-lag window only.

Deliberately NOT a required context

released-line-audit is schedule/dispatch only, so it can never report on a PR -- making it required
would be the required-but-absent trap. It is advisory by placement, not by continue-on-error: it
goes red on a finding, and tests/test_security_posture.py holds that property.

It also could not be actioned by a PR author: the remedy is cutting a release on the latest line
(docs/SUPPORT-POLICY.md), not editing the branch under review. The only change to
.github/required-contexts.txt is the comment recording this reasoning -- no context is added, and
the live required set stays at 13.

Pre-merge self-tests, all four run and passed

test result
A pip-audit -r against the v0.3.2 core lock exit 1, names PYSEC-2026-3552 on cryptography 49.0.0
B pip-audit -r against the origin/main lock exit 0 -- proves it DISTINGUISHES, not merely that it can go red
C empty lock, 0 pinned against a floor of 25 fail-closed path fires
D tag selector exactly v0.3.2; webconsole-v0.2.15 excluded

B is the one that matters: a gate that can only go red is not a gate.

Local verification

ruff check + format clean; 179 passed / 8 skipped across the workflow/backlog/ledger/posture
selection; ledger_check --ci clean; backlog_status_check OK; full pre-commit set passed including
actionlint. No --no-verify. mypy on tests/ shows only pre-existing dict type-arg errors also
present on origin/main (the gate's scope is mypy messagefoundry).

Not done, needs main

Self-tests E-G: dispatch security.yml on main and confirm released-line-audit goes RED on v0.3.2
printing the candidate list, selection rule, 41 pinned requirements and the advisory; confirm the
06:00 UTC cron opens an issue titled "Nightly Security is failing"; and after a release carrying
cryptography 50.0.0, confirm it goes GREEN and the issue closes.

If the notice never appears, the gate is reporting into the void. That is the one thing worth
chasing post-merge.

A follow-up to docs/SUPPLY-CHAIN.md is deferred because #264 edits that file; #1081 records the
deferred half.

…daily audits do not cover (BACKLOG #1081)

The `pip-audit` gate audits the CHECKED-OUT tree, so on the daily cron it audits `main`. "Is what we
would ship next current" is a different question from "does the version we already shipped carry a
known advisory", and the two answers diverge from the moment a fix lands on main until a release
carries it. Nothing watched that window. `released-line-audit` does, on the existing daily cron.

WHAT THIS IS NOT. The framing that motivated this -- "advisories disclosed after a tag are never
re-evaluated" -- was WRONG and is retracted in #1081. CVE-2026-69247 was caught the day it published,
by an existing required gate: commit ac87246 records "pip-audit (a required gate) flagged
cryptography 49.0.0 for CVE-2026-69247". Nor was main ahead of the tag; both read 49.0.0 at that
moment. Detection was never missing. Only the release-lag window was unwatched.

THE INSTRUMENT WAS PROVEN BEFORE THE GATE WAS WRITTEN, because a gate nobody proved can see its own
defect class is the failure this repo keeps re-learning. Against v0.3.2's core lock `pip-audit` exits
1 naming PYSEC-2026-3552 on cryptography 49.0.0; against origin/main's it exits 0 -- so it
distinguishes the two states rather than only ever reddening, which is the false-deny direction no red
test surfaces. An empty lock reads 0 pinned requirements against a floor of 25 and takes the
fail-closed path. The tag selector returns exactly v0.3.2 and excludes webconsole-v0.2.15. The
positive control is durable: the vulnerable lock is in git history, so a dispatch with
released_line_audit_tag=v0.3.2 re-arms it forever.

NO VEX IS APPLIED TO THIS GATE, and that is the load-bearing design decision. security/vex/README.md's
own worked example names the product with no version qualifier, so a `fixed` or `not_affected`
statement written on main would suppress the finding against the ALREADY-SHIPPED release -- turning
the gate green at the exact moment the assessment was written and before any release carried the fix.
`--ignore-vuln <ID>` is the escape hatch instead: explicit, per-advisory, greppable.

Advisory by PLACEMENT, not by continue-on-error -- the dast.yml posture. It is schedule/dispatch-only
so it can never report on a PR and must never be required (the required-but-absent trap), but it still
goes red on a finding. tests/test_security_posture.py could not express that: it had two buckets and
assumed anything not required carries continue-on-error. It now has three, and the new bucket asserts
the opposite -- no continue-on-error, not required, and never reachable from a pull_request.

nightly-notice.yml now watches `Security` as well as `CI`, because a red schedule-only job otherwise
reports nowhere. Its issue title is DERIVED from the completed workflow's name rather than hardcoded,
so a green nightly CI cannot close an issue a red Security run opened; for CI the title is
byte-identical to today's, so any currently-open issue stays matched.

Also corrects docs/CI.md, which said the only non-required security.yml jobs are `sbom` and `trivy`
and that both declare continue-on-error. Three jobs now sit outside the required set, in two different
ways, and the new one deliberately declares no continue-on-error.

Deferred deliberately: the docs/SUPPLY-CHAIN.md half -- scoping "continuously audited by pip-audit" to
main's lockfiles, and the releases/latest/download permanent fetch URLs. PR #264 edits that same file;
stacking the two would risk a conflict. #1081 records it, along with at least five other things left
unbuilt and why.

Self-tests A-D above were run pre-merge. E-G (live dispatch, notifier, recovery) need this on main and
are listed in #1081.
@wshallwshall
wshallwshall enabled auto-merge (squash) August 7, 2026 05:22
…ite (BACKLOG #1081)

`test_lock_installed_toolchain_install_is_hash_verified` pins the EXACT number of
`pip install --require-hashes -r ci/locks/ci-scanners.lock` sites per workflow, not just "at least
one". 73dccc3 added a third to security.yml (the released-line-audit toolchain install) and left the
constant at 2, so the test failed -- correctly. Verified the count directly rather than trusting the
message: sites are at security.yml lines 116, 219 and 483.

The exactness is the point, and the comment above the constant says why: a non-vacuity check satisfied
by one surviving line lets either half of a pair be deleted while every check stays green, and the
coverage job's failure mode is silent (`|| true` swallows the missing plugin). So the fix is the
number, not the assertion.

Updated that comment too. It enumerated the sites as "twice (the pip-audit step and the bandit step)";
bumping the constant while leaving the prose saying twice would be the same code-versus-prose drift
BACKLOG #1081 exists to close.

Caught by the coordinator on PR #267, not locally: the pre-commit run for 73dccc3 used a deliberately
scoped `-k` selection that did not include this file. A scoped selection cannot report the absence of a
failure outside its scope, and its green reads identically to the suite's green.
@wshallwshall
wshallwshall merged commit e18fb2c into main Aug 7, 2026
36 checks passed
@wshallwshall
wshallwshall deleted the claude/released-line-audit branch August 7, 2026 21:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant